External quality assessment to support the WHO ProSPeRo study for the evaluation of two dual HIV/syphilis point-of-care tests in seven countries

Background Sexually transmitted infections (STIs) such as syphilis and HIV remain to be a significant public health issue worldwide. Dual rapid point-of-care tests (POCTs) have shown promise for detecting antibodies to HIV and syphilis but have not been fully evaluated in the field. Our study supported the WHO ProSPeRo study on Sexually Transmitted Infection Point-of-Care Testing (STI POCT) by providing external quality assessment (EQA) for HIV and syphilis testing in reference laboratories and their associated clinical sites in seven countries. Methods HIV/syphilis serum liquid and dried tube specimen (DTS) panels were prepared by CDC. Liquid panels were distributed to the reference laboratories for three rounds of testing using commercially and locally available laboratory-based serological tests. DTS panels were sent to the clinical testing sites for 8 rounds of POC testing using the Abbott SD BIOLINE HIV/Syphilis Duo test (hereafter referred to as SD BIOLINE) and the Chembio Dual Path Platform (DPP) HIV-Syphilis assay. EQA panels were tested at CDC using the Rapid Plasma Reagin (RPR) test and the Treponema pallidum Particle Agglutination assay (TP-PA) for syphilis antibodies. Genetic Systems HIV-1/HIV-2 Plus O EIA, Geenius HIV Supplemental Assay and the Oraquick Advance HIV test were used to detect HIV antibodies in the EQA panels. Results from the reference laboratories and POCT sites were compared to those obtained at the CDC and a percentage agreement was calculated. Results Qualitative RPR and TP-PA performed at the reference laboratories demonstrated 95.4–100% agreement with CDC results while quantitative RPR and TP-PA tests demonstrated 87.7% and 89.2% agreement, respectively. A 93.8% concordance rate was observed for qualitative HIV testing in laboratories. EQA testing at clinical sites using dual tests showed 98.7% and 99.1% agreement for detection of HIV antibodies and eight out of 10 sites had > 95.8% agreement for syphilis testing. However, two clinical sites showed only 65.0–66.7% agreement for SD BIOLINE and 84.0–86.7% for DPP, respectively, for syphilis testing. Conclusions Overall, laboratories demonstrated high EQA performance in this study. Both HIV/syphilis POCTs gave expected results in the clinic-based evaluations using DTS. However, testing errors were identified in a few testing sites suggesting the necessity for continuous training and monitoring the quality of POC testing.


Background
Sexually transmitted infections (STIs) continue to be a major public health issue worldwide, particularly in under-resourced settings.More than 1 million new STIs cases are acquired every day, the majority of which are asymptomatic [1].Syphilis is an infection caused by the bacterium Treponema pallidum subspecies pallidum (T.pallidum).In 2020, the World Health Organization (WHO) estimated that 7.1 million new syphilis infections occurred worldwide [2].The human immunodeficiency virus (HIV) attacks the body's immune system and, if left untreated, it can lead to acquired immunodeficiency syndrome (AIDS).At the end of 2021, an estimated 38.4 million people worldwide are thought to be living with HIV and an estimated 1.5 million people recently acquired HIV infection [3].Both syphilis and HIV can be passed from mother to child.Approximately one million pregnant women (PW) have active syphilis globally each year, resulting in over 600,000 cases of congenital syphilis and greater than 350,000 adverse birth outcomes [4].In 2015, more than 1.4 million pregnant women were infected with HIV, and mother-to-child transmission (MTCT) of HIV is estimated to have resulted in over 150,000 infant cases [5].In 2014, WHO called for elimination of motherto-child transmission (EMTCT) of HIV and syphilis [6].
HIV and syphilis co-infection is evident in the United States and many other developed societies with robust surveillance systems.Both epidemics predominantly affect men who have sex with men (MSM) and other atrisk populations, such as sex workers and persons who use drugs [7,8].Untreated HIV infection may modulate the clinical presentation of syphilis and increase the risk for neurologic complications and treatment failure [9].Additionally, syphilis facilitates transmission and acquisition of HIV [10] and has a transient negative impact on the course of HIV infection with increased viral load and a decrease in the CD4 cell count [11,12].Concurrent syphilis infection in pregnant women living with HIV is significantly associated with vertical perinatal HIV transmission [13].The substantially higher economic burden and lower survival rate associated with late entry into medical care make early diagnosis essential [14].Therefore, rapid, accurate and easy-to-use dual HIV/syphilis tests provide simultaneous screening for the two diseases and could therefore substantially benefit patients and reduce the economic burden to health services.This is especially critical for pregnant women and key populations including men who have sex with men (MSM), sex workers and persons who use drugs.
Serological testing has long been the primary tool for diagnosing both syphilis and HIV infection.In the laboratory, both HIV antigens and antibodies can be identified using serological testing [15].A diagnosis of syphilis requires the use of both treponemal tests (detection of antibody to T. pallidum-specific proteins) and nontreponemal tests (detection of antibodies against lipoidal antigens from damaged host cells and treponemes) to confirm active or past/treated syphilis infection.
Introduction of single, disease-specific point-of-care tests (POCTs) has been instrumental in scaling-up both HIV and syphilis testing, especially in resource limited settings.In 2019, the rapid dual HIV/syphilis test was recommended by WHO as the first-line test for pregnant women attending antenatal care [16].The Abbott SD BIOLINE HIV/Syphilis Duo test (Abbott Point of Care Diagnostics, Princeton, NJ, hereafter referred to as SD BIOLINE) and the Chembio Dual Path Platform (DPP) HIV-Syphilis assay (Chembio Diagnostics, Medford, NY, hereafter referred to as Chembio DPP) are both singleuse rapid, qualitative, multiplex lateral-flow immunoassays for the detection of antibodies to HIV-1/2 and T. pallidum simultaneously.Chemio DPP was the first dual rapid test approved by FDA in 2020 and also received Clinical Laboratory Improvement Amendments (CLIA) waiver from FDA in 2023.SD Bioline HIV/Syphilis Duo test has been awarded WHO prequalification, making it the first dual HIV/syphilis point-of-care test available for deployment in national screening programs targeting resource-limited countries.Accurate results of rapid testing at point-of-care have a direct impact on establishment of a diagnosis, provision of treatment and prevention of further disease spread.However, there are limited data on the performance of dual HIV/syphilis tests in field settings [17,18].The global ProSPeRo study (Project on Sexually Transmitted Infection Point-of-care Testing) was established to address the need for standardized high-quality evaluation of STI POCTs which is critical for the further development and global uptake [19,20].This study, an integral part of ProSPeRo, was designed to assess the need for external quality assessment (EQA) for HIV and syphilis serological testing in both reference laboratories and their associated clinical sites that were participating in a stringent evaluation of dual HIV/syphilis POCTs in seven countries.

Participating laboratories and clinical sites
The dual HIV/syphilis EQA program was implemented in seven countries between 2018 and 2020.Reference laboratories that were not already taking part in an international EQA program and all ProSPeRo study sites were invited to participate in this study.Partici

Preparation of serum liquid and DTS panels
EQA liquid and dried tube specimen (DTS) panels were prepared at the STD Laboratory Reference and Research Branch (STDLRRB) of the US Centers for Disease Control and Prevention (CDC) following published methods [21,22].Briefly, HIV or syphilis reactive and non-reactive serum samples with volumes ranging from 200 to 700 ml were purchased from commercial sources (Plasma Service Group, PA, USA and Physician's Plasma Alliance, TN, USA) and tested at CDC to confirm the reactivity for HIV and syphilis as described below.Each EQA liquid and DTS panel consisted of 5 serum specimens: one syphilis reactive only, one HIV reactive only, one reactive for both syphilis and HIV, one non-reactive for both syphilis and HIV and one biologically false positive (BFP) syphilis serum.A BFP was defined as a serum sample that was reactive with a nontreponemal test but nonreactive with treponemal tests [23].To prepare reactive sera for both syphilis and HIV, syphilis and HIV reactive specimens were mixed in a 1:1 ratio and samples were tested to confirm expected results at CDC.
For the preparation of liquid EQA panels [22], 1.5 ml of each sample were dispensed into sterile glass vials under a laminar flow hood.Vials were then sealed, labelled, and stored refrigerated until shipped.For the preparation of DTS panels [21], each serum specimen was mixed with 0.1% (v/v) of green food colouring dye (McCormick, Hunt Valley, MD), 20 μL of the solution was dispensed into 2 ml plastic vials and left open in a laminar flow hood overnight for drying.Vials were then capped, labelled, and stored refrigerated until shipped.

Quality control (QC) and reproducibility testing of EQA panels at CDC
Five serum specimens included in the liquid and DTS EQA panels were tested at CDC using the qualitative and quantitative Rapid Plasma Reagin test (RPR, Arlington Scientific, Springville, UT) to detect nontreponemal antibodies and the T. pallidum Particle Agglutination assay (Serodia TP-PA, Fujirebio, Malvern, PA) to detect T. pallidum specific treponemal antibodies, following the manufacturers' instructions.EQA panels were also tested using Genetic systems HIV-1/HIV-2 Plus O EIA (Bio-Rad Laboratories, Hercules, CA), Geenius HIV Supplemental Assay (Bio-Rad Laboratories) and Oraquick Advance HIV test (OraSure Technologies, Bethlehem, PA) to detect HIV antibodies.To assess the reproducibility of rapid tests on DTS EQA panels, the panels were reconstituted and tested (described below) using two FDA-cleared syphilis single or dual rapid tests, Syphilis Health Check (SHC, Diagnostics Direct, Stone Harbor, NJ) and Chembio DPP, following the manufacturers' instructions for five consecutive days by the same tester.

Testing of liquid/DTS panels at laboratories and POCT sites
Each liquid/DTS EQA panel contained 5 vials of sample labelled A1 to A5, instructions for handling and testing and a result recording form.Each DTS panel also included one vial of reconstitution PT buffer containing 0.1 M Phosphate Buffered Saline (PBS) with 0.1% Tween-20 (pH 7.4, Sigma) and two plastic pipettes.Liquid panels were tested by laboratories using commercially and locally available laboratory-based serological tests.Both TP-PA/TP-HA (Treponema pallidum Haemagglutination) and RPR tests were qualitatively and quantitatively reported.DTS panels were tested at associated clinical sites using both SD BIOLINE and DPP dual HIV/syphilis tests.The day before testing, each DTS vial was reconstituted by adding 4 drops (200 µl) of PT buffer, mixed well and incubated overnight at room temperature to allow solubilization of dried serum into the PT buffer.On the test day, DTS panel sample tubes were mixed gently by tapping and subsequently tested following the manufacturers' instructions for serum/plasma specimen type.

Implementation of EQA
Both liquid and DTS EQA panels were shipped from -CDC to the laboratories in each participating country and DTS panels were then distributed by reference laboratories to the clinical POCT sites (Fig. 1).The laboratories conducted three rounds of EQA testing using liquid panels over a one-year period and 2-3 testers at each of the clinical POCT sites performed eight rounds of EQA testing using DTS panels over a 16-month period.All EQA panels were color coded and labelled appropriately to reflect the round number.Five serum specimens in each EQA panel were randomized for each round of testing and all sites received the same panel for each round.All test results from laboratories and POCT sites were returned to CDC for grading.

Grading reference laboratory test results
Laboratory test results for detection of HIV and T. pallidum antibodies were compared to results obtained at CDC STDLRRB.Each round of testing was assigned a maximum score of 100 points for 5 specimens (20 points for each specimen).The passing score for each round was 80.The overall grade was an average of all the tests performed by each laboratory.For HIV and syphilis qualitative tests, an incorrect result was scored 0. For quantitative RPR and TP-PA tests pertaining to reference labs, results were expressed as a titer (the reciprocal of the highest dilution at which the tested sample produced a reactive result).A quantitative result was scored 20 points if titers were within one dilution (two-fold) above or below expected results; scored 10 points if outside one dilution but within two dilutions (fourfold); scored 0 points if the result was outside two dilutions of the expected value.

Grading POCT site test results
Each DTS panel containing 5 samples was tested by 1-3 testers at each POCT site.Each dual test (SD BIOLINE and DPP) generated results for both HIV and syphilis.Therefore, there were four test results for each sample in the DTS panel: Chembio syphilis, Chembio HIV, SD Bioline syphilis and SD Bioline HIV.Each correct result was assigned 5 points, thus readings for each specimen could yield maximum of 20 points.Thus, each round of 5 specimens could score a maximum of 100 points.The score of each round was taken as the average score of all testers.The score for each site was the average score of all test rounds conducted at that site.

Statistical analysis
All results were analyzed using SPSS (IBM ® SPSS for Windows, Version 21.0) and R software [24].Results from laboratories and clinical POCT sites were compared to those obtained by CDC, which served as the reference standard.The percentage agreement was calculated as following: % agreement for reference labs = (total sample number-qualitative incorrect or titer incorrect sample number)/total sample number × 100% agreement for POCT sites = (total sample number -false positive-false

QC results of samples included in the EQA panel
The QC results obtained with 5 serum samples (A1-A5) included in the EQA panel were summarized in Table 1.Sample A1 was HIV reactive, but non-reactive for both treponemal and nontreponemal antibodies.Sample A2 was HIV non-reactive but reactive for treponemal antibodies.The RPR was also reactive for sample A2 with a titer of 1:2.Sample A3 was a negative control and had no detectable antibodies to both HIV and syphilis.Sample A4 was reactive for both HIV and syphilis with an RPR titer of 1:1.Sample A5 was a biological false positive serum showing non-reactivity to HIV and treponemal antibody testing but was reactive by RPR testing (titer 1:4).The reproducibility study of EQA DTS panel demonstrated consistent results during 5 consecutive days of testing at CDC STDL-RRB using SHC and Chembio tests (data not shown).There were a total 592 to 607 vials of DTS produced for each sample at CDC for this study (Table 1).

Participating laboratories and POCT sites
From 2018 to 2020, laboratories (Labs 1-7) in seven countries including Morocco, South Africa, Uganda, Italy, Malta, United Kingdom (UK) and Peru from three continents (Africa, Europe and South America) participated in this study.Four out of seven laboratories completed all three rounds of testing.One laboratory (Lab1) was only able to complete one round while two laboratories (Lab6 and Lab7) did not perform any rounds of EQA to date as they already participated in international EQA schemes (Table 2).Overall, 15 rounds of testing (3 rounds per laboratory for Lab 1-5) were expected from all laboratories, of which 13 rounds were performed with a participation rate of 86.7%.The number of clinical POCT sites recruited in each country varied from 1 to 4. Eight rounds of testing with 3 testers per site were expected from each POCT site; however, the actual number of EQA rounds in each site as well as the number of testers for each round varied from 0 to 8 across the EQA program (Table 1).Ten out of 14 sites (71.4%) performed at least one round of testing.A total of 50 rounds of testing were performed on all sites.A total of 140 reports from 14 POCT testing sites were submitted for review.

EQA performance of laboratories
Five laboratories (Lab1-5) participated in the EQA program (Table 2).Four laboratories completed three rounds of testing.Lab1 performed round 1 testing and did not participate in EQA round 2 and 3. A total of 13 rounds of testing was completed by laboratories with 5 specimens for each round and a total of 65 specimens in total.
As shown in Fig. 2, all laboratories scored above 80 for each test round which resulting in a 100.0%pass rate.The average scores for Lab1 to Lab5 were 93.3, 93.3, 88.0, 91.7 and 97.5, respectively.As shown in Table 3, for syphilis testing, 3 samples had discordant results for qualitative nontreponemal RPR tests when compared to the results from CDC.The overall agreement for qualitative RPR testing was 95.4% with a kappa value of 0.91, which suggests very good agreement.For RPR quantitative testing, 8 discordant results were observed in the laboratories, and all had titers ≥ fourfold higher as compared to reference results.The overall agreements were 87.7% and kappa value was 0.76, suggesting moderate agreement.
Results for qualitative treponemal TP-PA/TP-HA testing from the laboratories had 100.0%agreement with reference results.There were 11 discordant results identified in all rounds of testing from laboratories for quantitative TP-PA/TP-HA testing and all had titers outside 2-dilutions compared to reference results, which resulted in 82.5% agreement and 0.46 kappa value.Lab5 did not determine the actual TPHA endpoint titer.For qualitative HIV testing, 4 samples gave discordant results demonstrating a 93.8% concordance rate and a kappa value of 0.90.

EQA performance of clinical POCT sites
Among 140 sample test results submitted from the sites, 132 results had a passing score (> 80) with a pass rate of 94.3%.The average score among all 10 sites ranged from 82.8 to 100.0 with an average of 96.2 (Fig. 3).There were 36 reports with one or more discordant results as Fig. 2 EQA scores for participating laboratories.EQA liquid panels (5 serum samples per panel) were tested at each participating laboratory (Lab1-Lab5) using the actual test kits and reagents that they routinely use in the laboratory.The overall score for each laboratory was an average of all the tests performed.Score for each round from each participating laboratory was listed in a bar chart  4).At site 3, there were 10 and 12 false negative syphilis test results using DPP or SD BIOLINE rapid test, respectively.This resulted in 86.7% agreement (for DPP) and 84.0%(for SD BIOLINE) between site 3 and CDC.There were 19 false negative and 1 indeterminate result using the DPP test for detection of T. pallidum antibodies at site 10, while 18 false negative and 3 indeterminate results were observed using the SD BIOLINE test for syphilis antibody detection.These led to 66.7% agreement for DPP and 65.0% for SD BIOLINE for syphilis testing between site 10 and CDC.For the detection of HIV antibodies in all POCT sites, most test results from DPP and SD BIOLINE were consistent with reference results with overall 98.7%/ 99.1% agreement and a 0.958 / 0.979 kappa value, respectively (Table 5).

Discussion
With the high global burden of HIV and syphilis, especially in pregnant women and key populations, introduction of quality POC testing is extremely important to bring the diagnostic testing close to the site where Fig. 3 EQA score for each round in clinical POCT sites.EQA DTS panels (5 serum specimens) were tested in each clinical POCT site using both Abbott SD BIOLINE HIV/syphilis Duo test and Chembio DPP HIV-Syphilis assay.Results from the POCT sites were graded using results from CDC as the reference method.An average score from all testers for each round in each site was listed in a bar chart  clinical care is delivered.Dual HIV/syphilis rapid testing has the additional advantage of detecting two infections simultaneously using the same specimen.WHO has recommended using dual HIV/syphilis rapid tests for pregnant women since 2019, not only as the first test in antenatal care to help countries achieve EMTCT for both HIV and syphilis, but also as an important cost-saving measure [25].New reduced costs of dual HIV/syphilis POCTs were recently announced to accelerate EMTCT progress [26].There are many challenges associated with POCT to ensure test quality.Successful participation in an established EQA program should provide objective evidence of the testing sites' competence for POC testing of specimens not only in laboratories, but also in clinical field POCT sites.If any technical and methodological problems are identified in EQA, it could help participating sites to improve the quality of testing.EQA for POCT sites could be organized and overseen by the local or national laboratories, who in their turn should prove their competence by participating in (inter)national EQA programs.
Overall, the reference laboratories that participated in the evaluation demonstrated good EQA performance and all labs passed the EQA syphilis and HIV testing.Nontreponemal test titers are useful for monitoring efficacy of treatment progression and disease status (past/ treated, relapse or active infection).Patients successfully treated for syphilis typically show a fourfold (two doubling dilutions) decline in nontreponemal antibody titer over time, while a fourfold increase indicates either treatment failure or reinfection [27].Lab1 and Lab2 had 3 and 5 discordant RPR results respectively, with all discordant RPR results having titers ≥ fourfold dilutions above expected results.Overall, TP-PA/TP-HA quantitative tests demonstrated 82.5% agreement with those of the CDC laboratory.There were 1 discordant TP-PA/TP-HA results for Lab1 and 5 discordant results for Lab3 and Lab4.All these TP-PA/TP-HA discordant results had titers outside 2 dilutions of reference results.Variability in nontreponemal and treponemal quantitative test results between different laboratories may have been due to test kits from different manufacturers being used and this has been reported in earlier studies [28,29].Therefore, it is recommended to use the same syphilis serologic test, preferably from the same manufacturer and same laboratory to monitor treatment response [27].In addition, different laboratory personnel reading the test may have accounted for the discrepant results.The findings in this study indicate that any technically competent reference laboratory which undertakes evaluations of new POC tests for HIV and/or syphilis, or supervises the initiation of a program to use POC tests for these diseases at remote sites be, themselves, subject to a national or international EQA program.Including samples with low RPR titers in the EQA panel was extra challenging.Our study showed that only one laboratory missed one specimen with a low-reactive RPR (titer 1:4).All other laboratories successfully detected specimens with low RPR titers (1:1-1:4).
Dried tube specimens (DTS) were previously demonstrated to be a simple and cost-effective method for preparing quality assurance materials for use in resourcelimited settings [21] and this method was adopted in our EQA program for the POCT sites.DTS were successfully distributed by reference laboratories to remote clinical sites, where the POC testing was ultimately performed by non-laboratory personnel.After appropriate training, the staff at the clinical sites were able to reconstitute DTS  [30].Lower syphilis sensitivities of 66.2% and 68.6% were reported respectively for SD BIOLINE and DPP in a field performance evaluation study [18].In our study, high false negative rates (15-31%) were observed for syphilis testing for both SD BIOLINE and DPP tests at site 3 and 10, which are located in the continent of Europe and Africa respectively (Table 3).However, these DTS panels performed well in eight other testing sites and results from the same sample were correctly detected for HIV antibodies on the same dual test device.These noted discrepancies seem to be site specific and not related to a specific tester at those two sites since errors mostly occurred in the same specimens among all testers.Our study has some limitations.DTS syphilis specimens were reported to produce expected results after storage at 2-8°C or at 18-24°C for up to 3 weeks and our panels likely fell within these temperature ranges during shipping to sites and storage [35].However, DTS panels were not tested for reproducibility of results at different temperature and humidity levels in this study.Because of the study design limitation, no repeat testing of prepared panels, or follow-up with the participating site was possible to troubleshoot noted testing discrepancy.Turnover of staff at laboratories and, more specifically, at some clinical sites resulted in the need for ongoing supervision and training.The testing of panels occurred in different geographical areas including countries in Africa, Europe and South America and therefore, the results cannot be generalized to other regions.Dual HIV-syphilis rapid tests are used as the first test for pregnant women as part of the antenatal care but should be integrated into the overall algorithm for HIV testing for final diagnosis as recommended by WHO.This EQA study did not examine the quality of the overall HIV algorithm for pregnant women.However, this study, undertaken in association with a broad-based evaluation of POCTs, provides important information that could help authorities to design POCT evaluations in the future and the need to create a laboratory infrastructure capable of supporting the roll-out of a successful, quality POCT program through appropriate training, technical support, and technology transfer.

Conclusions
Our external quality assessment to support the WHO ProSPeRo study for the evaluation of two dual HIV/syphilis point-of-care tests was performed in reference laboratories and their associated clinical sites in seven countries using HIV/syphilis serum liquid and DTS panels.Overall, laboratories demonstrated high EQA performance in this study.Both HIV/syphilis POCTs gave expected results in the clinic-based evaluations.DTS can be a useful and robust tool to monitor the quality of POC testing.However, the quality of testing varied at laboratories and sites suggesting the necessity for stringent quality assurance programs for POC testing.EQA results can be used to identify laboratories/sites where staff require continuous training.

Fig. 1
Fig. 1 Schematic of EQA implementation workflow.Liquid and DTS EQA panels were prepared at CDC STDLRRB.All panels were shipped to reference laboratories who then distributed DTS panels to clinical POCT sites.All test results were sent back to CDC STDLRRB for grading

Table 1
QC results of samples included in the EQA panel RPR Rapid Plasma Reagin, TP-PA T.pallidum particle agglutination assay, Genetic System Genetic systems HIV-1/HIV-2 Plus O EIA, Geenius Geenius HIV Supplemental Assay, Oraquick Oraquick Advance HIV test, DTS Dried tube specimen, SYP Syphilis, R Reactive, NR Non-reactive, BFP Biologically false positive

Table 2
Participating laboratories and POCT sites

Table 3
EQA performance for each test type in laboratories a Specimen samples that did not have actual end-point titers or on which no testing occurred by lab were excluded from the % agreement and Kappa calculations

Table 4
Performance of dual HIV/syphilis test for detection of T. pallidum antibodies at POCT sites

Table 5
Performance of dual HIV/syphilis test for detection of HIV antibodies at POCT sites [31][32][33][34]][32][33][34].One BFP syphilis specimen was included for each EQA panel in this study and results were interpretated correctly at all POCT sites, showing no interference with the dual HIV/syphilis rapid tests at all POCT sites.Overall, DTS offered a reliable form of EQA for HIV and syphilis POC testing in remote settings in this study.